Artificial intelligence guidance system for robotic surgery

ABSTRACT

This invention is a system and method for utilizing artificial intelligence to operate a surgical robot (e.g., to perform a laminectomy), including a surgical robot, an artificial intelligence guidance system, an image recognition system, an image recognition database, and a database of past procedures with sensor data, electronic medical records, and imaging data. The image recognition system may identify the tissue type present in the patient and if it is the desired tissue type, the AI guidance system may remove a layer of that tissue with the end effector on the surgical robot, and have the surgeon define the tissue type if the image recognition system identified the tissue as anything other than the desired tissue type.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/691,023, filed Mar. 9, 2022, entitled “Artificial IntelligenceGuidance System for Robotic Surgery”, which is a continuation of U.S.patent application Ser. No. 17/097,328, now U.S. Pat. No. 11,304,761,filed Nov. 13, 2020, entitled “Artificial Intelligence Guidance Systemfor Robotic Surgery”, which is a continuation of U.S. patent applicationSer. No. 16/582,065, now U.S. Pat. No. 10,874,464, filed Sep. 25, 2019,entitled “Artificial Intelligence Guidance System for Robotic Surgery”,which is a continuation of U.S. patent application Ser. No. 16/288,077,filed Feb. 27, 2019, now U.S. Pat. No. 10,517,681, entitled “ArtificialIntelligence Guidance System for Robotic Surgery”, which claims thebenefit of U.S. Patent Application No. 62/636,046, filed Feb. 27, 2018,entitled “Artificial Intelligence Guidance System for Robotic Surgery,”each of which is incorporated by reference herein in its entirety forall purposes.

FIELD OF THE DISCLOSURE

The present disclosure is generally related to robotic surgery and morespecifically to surgery utilizing artificial intelligence to operate asurgical robot (e.g., to perform a laminectomy), further including anartificial intelligence guidance system leveraging an image recognitionsystem.

BACKGROUND

As far back as 3,500 years ago, Egyptian doctors were performinginvasive surgeries. Even though our tools and knowledge have improvedsince then, until very recently surgery was still a manual task forhuman hands.

About 15 years ago, Intuitive Surgical's da Vinci surgical robot was anew surgery device, that is routinely used to help surgeons be moreprecise, especially to remove natural hand tremors during surgery.

Since Intuitive Surgical's da Vinci surgical robot arrival, there havebeen many other surgical robots introduced. Today we are in a new waveof innovation that is best characterized by the convergence of surgicalrobotics with artificial intelligence (AI) and data gathered fromrobotic systems. We are now “digitizing” surgery by collecting andanalyzing data passing through these robotic systems, such as in-motiontracking, capturing images, etc. This then allows for enhancements tothe surgical processes.

For example, minimally invasive spine surgery has recently been advancedwith the use of endoscopes, with innovations in the imaging equipmentand advances in medical robotics. Advantages thankfully are to thepatient with less pain, smaller incisions, fewer complications and rapidreturn to normal activity as compared to conventional surgery. Surgeonsare now able to remove a ruptured disc with a very small endoscope andrepair a painful disc with the aid of a miniature camera and incisionsno larger than 0.5 inch. Robotics and computers are now playing anexpanding role in assisting the surgeon in these minimally invasiveprocedures where the surgeon sits at a station peering at a monitor thatshows a magnified view of the surgical field. A computer mimics andenhances the surgeon's hand movements. The computer in this instancemakes the movements more precise by dampening even a tiny tremor in thesurgeon's hands, which might increase the difficulty in performingprocedures under high power microscopic magnification. Even with therobot enhancing the surgeon's ability, a great deal of practice isrequired to master the technique.

Robots are also used to help in performing tasks that may be fatiguingfor surgeons. This idea formed “AESOP” which is a natural languagevoice-activated robotic arm that holds the camera and endoscope assemblyfor the surgeon during an endoscopic procedure. This innovation reducesthe need for a person to be required to do this task and improves theresult by moving precisely where the surgeon commands the robot,providing a steady image.

Computers are also being used in image guidance systems to give thesurgeon real-time images and allow him to navigate to the specificlocation on the spine. The surgeon can use digital information obtainedbefore surgery such as MRI or CAT scans or use real-time fluoroscopicx-rays to develop a three-dimensional image of the spine with the exactlocation of a probe placed on the spine. This technology has been shownto minimize errors in placement of pedicle screws that are sometimesused to fix the spine. It is also expected that this technology willexpand to allow more precise targeting of the problem with minimalincisions and fewer surgical complications.

The use of robotics and computers in minimally invasive spine surgeryhas resulted in more accurate surgical procedures, shortened operativetime and fewer complications. It is expected that Computer-EnhancedImage Guidance Systems will improve the precision of these proceduresbecause of real-time 3-D imaging at the time of the surgery. Diagnosticstudies will be digitally transmitted to the operating room andprojected to monitors to further aid the surgeon in performing thecorrect procedure with minimal trauma to the patient.

Today there are basically three types of AI used for surgery. The firstis by IBM in its Watson System, which uses an expert-system type of AI.Watson stores vast medical information and gives responses to naturallanguage queries from surgeons. Watson becomes an intelligent surgicalassistant.

Second is “machine learning” algorithms. These algorithms useunsupervised pattern matching algorithms that would aid doctors inrecognizing when a sequence of symptoms results are matched to a similarpattern of a particular previous surgical issue or result. This willhelp surgeons have a learning machine at their side.

Third are technologies like “AlphaGo” that trains itself by taking dataand training itself to find its own patterns. All the surgical data andoutcomes are created and AlphaGo will do the surgeries virtually itselfto see if it can first replicate results and then later improve results.

Traditional methods of robotic surgery have not yet embraced AI inspecific areas, such as spinal surgery where AI is being leveraged forimage recognition through the procedure. Therefore, novel methods areneeded to leverage artificial intelligence to improve outcomes forrobotic surgery, such as minimally invasive robotic spinal surgeryprocedures.

The subject matter discussed in the background section should not beassumed to be prior art merely because of its mention in the backgroundsection. Similarly, a problem mentioned in the background section orassociated with the subject matter of the background section should notbe assumed to have been previously recognized in the prior art. Thesubject matter in the background section merely represents differentapproaches, which in and of themselves may also correspond toimplementations of the claimed technology.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various embodiments of systems,methods, and embodiments of various other aspects of the disclosure. Anyperson with ordinary skills in the art will appreciate that theillustrated element boundaries (e.g. boxes, groups of boxes, or othershapes) in the figures represent one example of the boundaries. It maybe that in some examples one element may be designed as multipleelements or that multiple elements may be designed as one element. Insome examples, an element shown as an internal component of one elementmay be implemented as an external component in another and vice versa.Furthermore, elements may not be drawn to scale. Non-limiting andnon-exhaustive descriptions are described with reference to thefollowing drawings. The components in the figures are not necessarily toscale, emphasis instead being placed upon illustrating principles.

FIG. 1 illustrates a robotic surgery system and a method of utilizingartificial intelligence, according to an embodiment.

FIG. 2 illustrates a Surgical Control Software module, according to anembodiment.

FIG. 3 illustrates an Incision Marking Module, according to anembodiment.

FIG. 4 Illustrates an AI Guidance System, according to an embodiment.

FIG. 5 illustrates a Progression Module, according to an embodiment.

FIG. 6 illustrates a robotic system, according to an embodiment.

FIG. 7 illustrates end effectors, according to an embodiment.

DETAILED DESCRIPTION

Some embodiments of this disclosure, illustrating all its features, willnow be discussed in detail. The words “comprising,” “having,”“containing,” and “including,” and other forms thereof, are intended tobe equivalent in meaning and be open ended in that an item or itemsfollowing any one of these words is not meant to be an exhaustivelisting of such item or items, or meant to be limited to only the listeditem or items.

It must also be noted that as used herein and in the appended claims,the singular forms “a,” “an,” and “the” include plural references unlessthe context clearly dictates otherwise. Although any systems and methodssimilar or equivalent to those described herein can be used in thepractice or testing of embodiments of the present disclosure, thepreferred, systems and methods are now described.

Embodiments of the present disclosure will be described more fullyhereinafter with reference to the accompanying drawings, and in whichexample embodiments are shown. Embodiments of the claims may, however,be embodied in many different forms and should not be construed aslimited to the embodiments set forth herein. The examples set forthherein are non-limiting examples and are merely examples among otherpossible examples.

Advancements in real-time image recognition systems (You Only Look Once:Unified, Real-Time Object Detection) make it possible to identify thetype of tissue in front of a surgical robot's end effector. For example,this inventive system may utilize one of many available surgicalplanning systems to chart out the steps in the procedure for the currentpatient. At each step of the procedure, the desired tissue to beaffected by the surgical robot's end effector(s) is defined in thesurgical plan. The end effectors can include, without limitation,robotic grippers, cutting instruments (e.g., scalpels), cannulas,reamers, rongeurs, scissors, drills, bits, or the like. The degrees offreedom, sizes, and functionalities of the end effectors can be selectedbased on the procedure to be performed. For example, one end effectorcan be used to cut and remove bone and another end effector can be usedto remove cartilage, discs, or the like. A series of end effectors canbe used to perform a surgical procedure according to the surgical plan.

The system may take an image of the point of interest (area to be workedon in this step in the surgery) and send that image through an imagerecognition system. If the desired tissue type is identified by thesystem, the progress through the surgical step may be calculated bycomparing the number of layers of tissue affected by the robot in thecurrent procedure to the average number of layers effected to completethis surgical step in statistically similar patients who had the sameprocedure. That progress is displayed for the surgeon, the tissue isaffected as prescribed in the surgical plan and the process repeatsuntil the desired tissue type is not identified by the image recognitionsystem. When the desired tissue type is not identified, the robot stopsits progress and the image is presented to the surgeon to define. If thesurgeon defines the tissue as the desired type, the identified imagelibrary in the image recognition database is updated and the robotproceeds.

In some embodiments, the system may obtain one or more images of aregion of interest, and the images can be sent to an image recognitionsystem. The images can be still images or video. If targeted tissue isidentified by the system, a surgical plan can be generated. For example,the targeted tissue can be identified by comparing the one or moreimages to reference images. The comparison can be used to identifytissue to be removed, determine when a procedure is completed, etc. Insome embodiments, the targeted tissue can be identified by comparing thenumber of layers of tissue affected by the robot in the currentprocedure to reference data (e.g., the average number of layers effectedto complete this surgical step in statistically similar patients who hadthe same or similar procedure). That progress is displayed for thesurgeon, the tissue is affected as prescribed in the surgical plan andthe process repeats until the targeted tissue has been removed. Therobot then stops its progress and the image is presented to the surgeonto define. If the surgeon defines the tissue as targeted tissue, theidentified image library in the image recognition database is updatedand the robot proceeds. This process can be applied to each individualstep in the spinal surgery process as detailed herein.

In certain embodiments, systems and methods can utilize artificialintelligence to operate one or more surgical robot systems, including asurgical robot apparatus, an artificial intelligence guidance system, animage recognition system, an image recognition database, and/or adatabase of past procedures with sensor data, electronic medicalrecords, and/or imaging data. The image recognition system may identifythe tissue type present in the patient. If it is the desired or targetedtissue type, the AI guidance system may remove that tissue using an endeffector on the surgical robot. The surgeon can define the tissue typeif the image recognition system identified the tissue as anything otherthan the desired tissue type to perform a procedure. The system canidentify anatomical features, abnormalities, tissue margins, tissuecharacteristics, tissue types, tissue interfaces, or combinationsthereof based on, for example, preset criteria, physician input, etc.For example, the image recognition system can evaluate images toidentify landmarks and generate a surgical plan based, at least in part,on those landmarks. The landmarks can be identified by the system,physician, or both. In some procedures, the landmarks can beidentifiable anatomical features (e.g., spinous processes, bonyprotrusions, facet joints, nerves, spinal cord, intervertebral disc,vertebral endplates, etc.) along the patient's spine to generate asurgical plan.

In certain embodiments, systems and methods can use images obtainedprior to and/or during surgery to guide a robotic surgical apparatus,end effector, surgical tool, or the like. Illustratively, an endoscopecan be used as a guide wire. The endoscope can constantly interact withthe anterior-posterior (AP) view, allowing a surgeon to be constantlylooking at the endoscope. This system can be expanded to cover theentirety of the surgical procedure. Using the endoscope to function as aguide wire allows for locating the endoscope inside of the patient as anadditional reference point for the surgical navigation program. Theconfiguration of the endoscope can be selected based on the instrumentto move delivered over it.

In certain embodiments, systems and methods can monitor a patient'sbrain activity during surgery to determine a level of consciousness,patient response during a procedure, or the like. For example, using ofa wireless EEG system during surgery can provide a basis for determiningthe amount of medication to give a patient. The EEG can track the amountof discomfort the patient is experiencing, and more medication (i.e.,anesthesia) can be administered if the amount of discomfort exceeds athreshold. The system can include an artificial intelligence unit thatreceive monitored brain activity data (e.g., brain activity patterns,brain activity spikes, etc.) and identify correlations with anesthesiabased adverse events. Pain, discomfort, and other patient parameters canbe monitored and evaluated to determine whether to modify the treatmentplan, administer anesthesia, etc. The AI/machine learning can be used toanalyze brain activity, patient feedback, or other patient parametersto, for example, improve safety, comfort, or the like.

In certain embodiments, systems and methods can include the measuring ofvarious parameters associated with an end effector before, during,and/or after a surgical action or procedure. The monitored parameterscan include rpms, angle, direction, sound, or the like. The monitoredparameters can be combined with location data, tissue type data, and/ormetadata to train an artificial intelligence system for guiding arobotic surgical tool to automatically perform a surgical action,procedure, or an entire surgery.

In some embodiments, a method implemented in a computing system for atleast partially controlling a robotic surgical apparatus to performsurgical actions by obtaining a first image of a region of interestassociated with a subject. A type of tissue shown in the first image canbe identified based, at least in part, on a neural network model trainedon an image training set. In response to determining that the identifiedtype of tissue belongs to a set of targeted types, causing the roboticsurgical apparatus to perform a first surgical action with respect tothe region of interest in accordance with a surgical plan. A secondimage of the region of interest can be obtained after completion of thefirst surgical action. Additionally surgical steps can be performed.

A computer-readable storage medium storing content that, when executedby one or more processors, causes the one or more processors to performactions including obtaining first image of a region of interestassociated with a surgery subject, and identifying a type of tissueshown in the first image based, at least in part, on a neural networkmodel. In response to determining that the identified type of tissuebelongs to a set of targeted types, robotic surgical apparatus performsa first surgical action with respect to the region of interest inaccordance with a surgical plan. A second image of the region ofinterest is obtained after completion of the first surgical action. Theactions can include displaying types of tissue comprises displaying oneor more boundary indicators for indicating at least one of targetedtissue to be removed, protected tissue, delivery instrument placement,or an end effector working space within the subject.

FIG. 1 illustrates a robotic surgery system and a method of utilizingartificial intelligence to complete specific steps in a minimallyinvasive surgery, according to an embodiment. The system may include asurgeon computer, a surgical robot, and a robotic surgery controlsystem.

The surgeon computer, which can also be a mobile device, may include adisplay and a surgical planning module.

The surgical planning module allows the surgeon to create a plan for arobotic surgery procedure that is based upon the medical imaging of thepatient, such as described in U.S. Pat. No. 7,338,526.

The surgical robot, such as described in U.S. Pat. No. 5,784,542, mayinclude at least one camera and multiple end effectors.

The robotic surgery control system may include surgical controlsoftware, surgeon controls, a display, an image recognition database, aprocedure database and a medical image database.

The procedure database can include medical records data, images (e.g.,pre-and post-surgical images), physician input, sensor data, or thelike. The images can include MRI or CAT scans, fluoroscopic images, orother types of images. The sensor data can be collected duringprocedures, etc. related to all procedures of this type. This databaseis queried by the surgical control for all medical imaging from thecurrent patient and by the progression module for data for all similarpatients who had the same procedure.

The image recognition database is populated by images taken by thesurgical robot cameras that are defined by the surgeons and updated witheach use of the system for greater accuracy. The surgeon controls areused for manual manipulation of the surgical robot, either to take overwhen the AI cannot proceed or to navigate the end effector to the pointof interest.

The surgical control software may include an incision marking module,and an AI guidance system that may include a progression module. Thesurgical control software begins when initiated by the surgeon.

The pre-operative plan, as constructed by the user using a system suchas the one described in U.S. Pat. No. 7,338,526, is retrieved from theprocedure database.

The system may then initiate the incision marking module which willensure the patient is properly positioned and the incision site ismarked. When the incision marking module is complete the AI guidancesystem may be initiated. The incision marking module may be designed tocover the steps in the spinal surgery between when the patient is placedon the table and when the AI guidance system makes the first incision.The module begins when it receives a prompt from the surgical controlsoftware. The incision location, in this example just above the L4vertebrae, is identified from the pre-operative plan. The system maythen capture an image of the patient to determine if they are properlypositioned on the operating table. If they are not, the surgeon orsupport staff are prompted for the necessary adjustment and a new imagemay be captured. This loop continues until the system is satisfied thatthe patient is properly positioned.

Next, the placement of the guide wire may be checked by the imagingsystem. This process loops in the same way as the patient positioning islooped. The surgeon or support staff are prompted for the necessaryadjustment to the guide wire placement and another image is taken untilthe system is satisfied that the guide wire is properly placed. In thisexample, we are using a traditional guidewire, but several additionalguide wire methods and systems are detailed in the attached additionalembodiments of this system. When the patient position and guidewireposition are correct, the system will mark the incision site.

The AI guidance system may utilize the camera to take an image of thepoint of interest and the progression module may compare that image tothe image recognition database to determine if the tissue present is thedesired tissue type that will allow the surgical robot to proceed. Theprogress through the tissue type is displayed based on the number oflayers of the current tissue removed as compared to the average numberof layers removed in other patients who had the same procedure and had asimilar anatomical volume of their surgical point of interest.

In this example, the step in the spinal surgery the robotic surgicalsystem is completing utilizing artificial intelligence is the boneremoval portion of a laminectomy.

When the surgeon reaches the point in their surgical plan during whichthe lamina is going to be removed, the surgical robot may move a boneremoval end effector to the point of interest on the patient's spine.

An imaging system connected to the image recognition software is in thesame location. It can be co-located on the same robot arm as the boneremoval end effector or on another mount that allows it a view of thepoint of interest. The imaging system may take an image of the point ofinterest, and the progression module will run. When the tissue type isconfirmed, the bone removal end effector removes a small layer oftissue. The imaging system repeats the process of tissue typeconfirmation, followed by the end effector removing another layer oftissue. This loop continues until the imaging system identifies adifferent tissue type, ideally indicating the bone removal step iscomplete and the nerve tissue below has been exposed.

The imaging system and progression module are initially trained using aneural network/machine learning. Using machine learning systems whichconstruct algorithms that can learn from and then make predictions onthe image data, which is a common task in machine learning. Suchalgorithms work by making image data-driven predictions through buildinga mathematical model from image input data. The image data is used tobuild the final model which usually comes from multiple datasets (inthis case, dataset of previous operations visual data with metadataassociated with the images from doctor articulated tissue types). Inparticular, three data sets (images, metadata of tissue type andmetadata of bone portions unfolding in the images over time) may be usedin different stages of the creation of the model. A user can input orchange metadata. For example, the metadata can include surgeon definedmetadata. In some embodiments, the metadata can be defined by AIsystems. In some embodiments, the metadata can include both user and AIdefined data.

The model is initially fit on a training dataset, which is a set ofexamples used to fit the parameters (e.g., weights of connectionsbetween “neurons” in artificial neural networks) of the model. The model(e.g., a neural net or a naive Bayes classifier) may be trained on thetraining dataset using a supervised learning method (e.g., gradientdescent or stochastic gradient descent). In practice, the trainingdataset often includes pairs of generated “input vectors” with theassociated corresponding “answer vector” (commonly denoted as thetarget). The current model is run with the training dataset and producesa result, which is then compared with the target, for each input vectorin the training dataset. Based on the result of the comparison and thespecific learning algorithm being used, the parameters of the model areadjusted. The model fitting can include both variable selection andparameter estimation.

Successively, the fitted model can be used to predict the responses forthe observations in a second dataset called the validation dataset. Thevalidation dataset provides an unbiased evaluation of a model fit on thetraining dataset while tuning the model's parameters. Validationdatasets can be used for regularization by early stopping: stop trainingwhen the error on the validation dataset increases, as this may be asign of overfitting to the training dataset. This simple procedure iscomplicated in practice by the fact that the validation dataset's errormay fluctuate during training, which would require added ad-hoc rulesfor deciding when overfitting has truly begun. Finally, the test datasetis a dataset used to provide an unbiased evaluation of a final model fiton the training data.

Once this trained dataset is built, the real-time images may be fed intothe system and as tissues are identified, the tissue types are annotatedvirtually over the real time images, with a % probability ofidentification. This allows the doctor to have an AI image recognitionassistant.

The system includes a failsafe that allows the surgeon on hand to stopthe process. Stopping the process may include a teaching step in whichthe surgeon defines the tissue type visible, to improve thefunctionality of the image recognition software.

The failsafe system provides historical data of many operations thatstores the amount of time (video) and the Virtual identified images onthe tissue. The tissues identified may be in a time sequence as theoperation proceeds. In a real-time operation, the sequence ofimage-recognized tissue (and the timing of getting to and through theserecognized tissues) is compared to the historical database. If thereal-time recognized tissues are correlated with the same sequence oftissues in the historical database, the system proceeds. However, if arecognized tissue does not appear in the sequence history, or if therecognized tissue appears earlier than expected, the fail system isalerted, which causes an alarm, with a virtual message over thenon-normal images.

There could be other fail-safe triggers, such as (1) the length of timebetween recognized tissues that are normal, (2) the probability of therecognition trending down, (3) the image quality starting to degrade,etc. In this way the failsafe system could have multiple processesrunning simultaneously.

When the AI guidance system completes a step in its entirety, it mayreturn to the surgical control software, which determines based on thepre-operative plan, if the procedure is complete. If the procedure iscomplete, the program ends.

If the program is not complete, the pre-operative plan is consulted todetermine if the next surgical step requires a different end effector.

End effectors in this scenario also include tools such as retractortubes and surgical hardware, in addition to the incision markers, boneremoval tools, skin/muscle fascia incision tools, etc. If a new endeffector is needed, the surgeon or support staff makes the hardwareadjustment before the system proceeds to the next step in thepre-operative plan. After the needed end effector/tool is put intoplace, or if the same end effector/tool from the previous step isappropriate, the system may go back to the AI guidance system until thenext surgical step is completed. This process continues to loop untilthe procedure is complete. To perform multiple procedures on a patient,the end effector can be replaced to begin another procedure. Forexample, one set of end effectors can be used to perform a laminectomyand another set of end effectors can be used to perform a stenosisdecompression procedure at a different level along the spine.

One skilled in the art will appreciate that, for this and otherprocesses and methods disclosed herein, the functions performed in theprocesses and methods may be implemented in differing order.Furthermore, the outlined steps and operations are only provided asexamples, and some of the steps and operations may be optional, combinedinto fewer steps and operations, or expanded into additional steps andoperations without detracting from the essence of the disclosedembodiments.

FIG. 2 represents the surgical control software that is part of therobotic surgery control system, according to an embodiment. The systembegins when it is engaged by the surgeon.

The pre-operative plan, as constructed by the user using a system suchas the one described in U.S. Pat. No. 7,338,526, is retrieved from theprocedure database.

The system may then initiate the incision marking module which willensure the patient is properly positioned and the incision site ismarked. When the incision marking module is complete, the AI guidancesystem is initiated.

The AI guidance system works through each step in the surgical process.In this example, we are focusing on the removal of the lamina from theL4 vertebrae, but the system is applicable to one or more steps in thespinal surgery process, from initial incision, port placement, retractordocking, lamina removal, disc removal and/or hardware insertion. The AIguidance system runs until an isolated surgical step is complete. Whenthe AI guidance system completes a step in its entirety, it returns tothe surgical control software, which determines based on thepre-operative plan, if the procedure is complete. If the procedure iscomplete, the program ends.

If the program is not complete, the pre-operative plan is consulted todetermine if the next surgical step requires a different end effector.End effectors in this scenario also include tools such as retractortubes and surgical hardware, in addition to the incision markers, boneremoval tools, incision tools (e.g., skin/muscle fascia incision tools),etc. If a new end effector is needed, the surgeon or support staff canmake the hardware adjustment before the system proceeds to the next stepin the pre-operative plan. After the needed end effector/tool is putinto place, or if the same end effector/tool from the previous step isappropriate, the system may go back to the AI guidance system until thenext surgical step is completed. This process continues to loop untilthe procedure is complete.

FIG. 3 represents the incision marking module that is part of thesurgical control software, according to an embodiment. The incisionmarking module is designed to cover the steps in the spinal surgerybetween when the patient is placed on the table and when the AI guidancesystem makes the first incision.

The module begins when it receives a prompt from the surgical controlsoftware. The incision location, in this example just above the L4vertebrae, is identified from the pre-operative plan.

The module may then capture an image of the patient to determine if theyare properly positioned on the operating table. If they are not, thesurgeon or support staff are prompted for the necessary adjustment and anew image is captured. This loop continues until the system is satisfiedthat the patient is properly positioned.

Next, the placement of the guide wire is checked by the imaging system.This process loops in the same way as the patient positioning is looped.The surgeon or support staff are prompted for the necessary adjustmentto the guide wire placement and another image is taken until the systemis satisfied that the guide wire is properly placed. In this example, weare using a traditional guidewire, but several additional guide wiremethods and systems are detailed in the attached additional embodimentsof this system.

When the patient position and guidewire position is correct, the systemmay mark the incision site. This can be done in many ways, includinghaving the surgeon mark the site as guided by a projection from thesystem.

FIG. 4 represents the artificial intelligence (AI) guidance system. Thesystem begins when it receives a prompt from the surgical controlsoftware. While this system may be utilized at each step in the surgicalprocess, the example here will be the removal of the lamina from the L4vertebrae.

The point of interest, in this example the L4 vertebrae, is identifiedfrom the procedure database. The camera(s) is focused on the point ofinterest.

The end effector is navigated to the point of interest.

Then the progression module is run, which may update the progress on therobotic surgery control system display and return if the tissue at thepoint of interest is the desired tissue type, in this example thedesired tissue type is bone. So, if the tissue type identified is notbone, the system stops, alerts the surgeon and polls for their input.

The surgeon will need to define the tissue type currently at the pointof interest. If the surgeon defines the current tissue type as thedesired tissue type, this updates the image recognition database and thesystem returns to the progression module with the updated imagerecognition definitions. If the surgeon defines the tissue as any othertype of tissue than the desired tissue type, the image definition isadded to the image recognition database and the number of layers removedof the desired tissue type for the current patient is recorded in theprocedure database.

FIG. 5 represents the progression module, according to an embodiment.The progression module is triggered by the AI guidance system when theimaging system and the end effector are at the point of interest on thecurrent patient.

An image of the point of interest is taken and an image recognitionsystem, such as described in “You Only Look Once: Unified, Real-TimeObject Detection”, is used to identify the tissue type present in theimage taken of the point of interest on the current patient. The imagerecognition system utilizes the image recognition database to identifythe tissue type and to store the definitions of tissue types found inimages as they are defined by surgeons using the system.

You only look once (YOLO) is a state-of-the-art, real-time objectdetection system. Prior detection systems repurpose classifiers orlocalizers to perform detection. They apply the model to an image atmultiple locations and scales. High scoring regions of the image areconsidered detections. YOLO uses a totally different approach in that itapplies a single neural network to the full image. This network dividesthe image into regions and predicts bounding boxes and probabilities foreach region. These bounding boxes are weighted by the predictedprobabilities. The model has several advantages over classifier-basedsystems in that it looks at the whole image at test time so itspredictions are informed by global context in the image. YOLO also makespredictions with a single network evaluation which makes it extremelyfast, more than 1000× faster than most prior art systems.

The real-time images may be fed into a “trained neutral network imagesystem” as described above, which uses this historical data to informthe YOLO system. The real-time images may be used to identify the tissuetype present in the image taken of the point of interest on the currentpatient. Unlike simply identifying the tissues types, which we havediscussed above by adding a Virtual tissue tag on the images, this YOLOsystem goes further, in that it can detect distances and positionsbetween the boundary boxes. In this way, tissue type will not only bedefined virtually over the real-time images, but virtual distances areoverlaid and can be highlighted when they are outside norms (again thesedistances of boundary boxes are pre-trained). The image recognitionsystem utilizes the historical image recognition database and YOLO toidentify the tissue type and their positions to provide real-timeaugmentation data to the surgeons using the system.

If the tissue type identified is not the desired tissue type for thesurgical robot to proceed with tissue removal, the module ends andreturns to the AI guidance system. If the tissue type identified is thedesired tissue type to proceed with tissue removal, data related to theidentified tissue type is retrieved from the procedure database. Forexample, if the current patient is having a laminectomy done on his L4vertebrae, the system will retrieve all data related to laminectomiesperformed on the L4 of any patient in the procedure database. Thoseresults are then filtered by the progression module based on the volumeof the current patient's L4 vertebrae. This is calculated based on themedical imaging data, such as the patient's pre-surgery MRI.

The volume of the L4 vertebrae in this example is calculated for all thepatient data retrieved from the procedure database. Patients for whomthe volume of their L4 vertebrae is, for example, within one standarddeviation of the volume of the current patient's L4 vertebrae are keptfor comparison. The average number of layers removed from the desiredtissue type (in this example bone) to complete this step in theprocedure for the filtered patients is then calculated.

The progress of the current procedure, number of layers removed as apercentage of the average layers removed in similar patients, isdisplayed on the robotic surgical system display. The module thenreturns to the AI guidance system.

There are many improvements on the basic inventive principle describehere in many of the areas of the process.

Improvements in “Incision localization/marking” are made such asPre-Operative Image. A user can input information for performingprocedures. The information can include, without limitation, targetedtissue, non-targeted tissue, critical tissue (e.g., tissue to beprotected or avoided), access paths, cutting/drilling paths, instrumentorientations (e.g., delivery instruments, surgical instruments, etc.),working spaces, safety barriers, hold spots, or the like. Theinformation can be used to determine or modify a surgical plan and canbe inputted via a touch screen, keyboard, or the like. A method of usingan image in which a sketch on the image indicates parts of the bone areto be removed. This is a freehand adjustment by the surgeon to thepreoperative plan, layered on top of medical imaging (MRI, CT, etc.).This adjustment to the surgical plan is transmitted to the AI surgicalrobot and it only removes the desired area, the surgeon supervises therobot during the procedure to take over/resume the operation ifnecessary.

Improvements in “Incision localization/marking” are made such asPre-Operative Image using Interactive User Interface. Similar to 1a,except the image received from the surgical robot is displayed on atouch screen/user interface inside the operating room and the surgeonsketches on the image which of the corresponding area of tissue issupposed to be removed. Other important areas can be identified (such asnerves) to warn the robot to stay away from sensitive areas. This isapplicable to all steps past this one in this process but is documentedhere as this is the first step in which the surgeon would mark out areasduring the procedure as opposed to during pre-operative planning.

“Incision localization/markings” are made as Pre-Operative Images on anactual image using Interactive User Interface. The system would deploygraphical tools, similar to power point, that allows the surgeon to drawshapes of different colors over the image. The shapes can be auto filledwith the suggested colors and meta-tags (e.g., distance depth, speed ofdrill, amount of dither, etc.). For instance, the system could allow thesurgeon in drawing mode to define the draw pen or mouse to be defined as“red, 1 mm deep, 100 rpm, +/−5 rpm”, where red would correspond todrill, 1mm deep at 100+/−5 rpm. In another area for instance, thesurgeon could have defined a yellow +0.5 mm which is a region that therobot is barred from running. One could image many other user interfacecontrols, such as (1) cutting or drilling paths, (2) degrees of safetybarriers along the cutting, (3) hold spots, (4) jump to another spots,etc. The surgeon would stand by during the procedure and can turn offthe machine at any time. The drill also has built-in safeguards. Forexample, it can detect if it's too close to a nerve (e.g., a facialnerve) and will automatically shut off.

Improvements in “Incision localization/marking” are made such asPre-Operative Image using Interactive User Interface to Resolve LatencyIssues. Like 1b, except the focus is on adjusting the field of view(zooming in) to define in greater detail where tissue barriers ofinterest are and what actions should be taken relative to thosebarriers.

Improvements in “Incision localization/marking” are made such asMultiple Imaging Systems for Problem Space Identification in SpinalSurgery. A method that combines multiple imaging systems to identify aproblem space in a patient's spine. An algorithm is applied to theimages to calculate the best incision location based on where theproblem space is located. This algorithm accounts for the surgicalprocedure being used when identifying the incision site.

Improvements in “Incision localization/marking” are made such as GuideWire Placement for Best Incision Site using a Robot. A method thatenables a robot to manipulate the guide wire (i.e. adjust its length).The robot reviews an image to place the guide wire based upon bestpractices, which is learned through image recognition, historical data,and other sources. This method includes placing an ink tip on the guidewire that marks the location for the best incision site. Thisinformation is stored in the database, which allows the robot to accessthis information in following procedures. This method would increaseefficiency, accuracy, and repeatability for locating incision sites.

Improvements in “Incision localization/marking” are made such as RoboticGuide Wire Placement through Surgeon Commands. A method that allowssurgeons to annotate where a robot should move or adjust to in order toplace the guide wire while locating an incision site. The robot canlearn where it is commanded to move and store the information in adatabase. The robot can access this database to use for referencesduring future procedures. This increases efficiency, accuracy, andrepeatability for locating incision sites.

Improvements in “Incision localization/marking” are made such asMulti-Shaped Guide Wire. The method to create guide wires that havedifferent, adjustable 2D shapes. This will allow the surgeon to pick themost applicable shape to use for different procedures or at a specificpoint in a procedure. The shapes can also be produced through thecombining of different guide wires. Guidewire shape would be determinedby AI using correlations between patient attributes, procedure type,wire shape, and postoperative outcomes.

Improvements in “Incision localization/marking” are made such as ImagingSystem Output Projection onto Patient's Skin. A method for accuratelyprojecting an imaging system output (MRI, CT, X-Ray, Fluoroscopy, etc.)onto the patient to show where different tissue types are locatedunderneath the skin. The projection would also include a projection ofthe guide wire to help the surgeon visualize the best point of incision.This increases the accuracy of the incision point. This can be done withhigh-speed projectors, or with an augmented reality display for thesurgeon. Alternate embodiments can include virtual reality headsets forincision placement.

Improvements in “Incision localization/marking” are made such asArtificial Intelligence for Optimal Trajectory and Incision Placementfor Spinal Surgery. A software that utilizes artificial intelligence todetermine the optimal trajectory and incision placement for any type ofspinal surgery (e.g., spinal fusion, decompression procedures, screwplacement, cage insertion, etc.). This method uses information about thesurgery to decide the trajectory and incision site, such as screw size,the angle the screw will be inserted at, and other information. Avirtual line is then drawn out from where the drill will be placedduring surgery.

Improvements in “Incision localization/marking” are made such asIncision Site Location Means based on Screw Placement Information. Ameans for marking the incision site for a spinal surgical procedure thatincludes information that cites where the screw needs to be placed,which was determined from a mathematical calculation. This informationincludes an image, which shows the projected incision site from analgorithm. This process will help make the incision site more accurateand the process for finding this site more repeatable, regardless of thepatient's anatomy.

Improvements in “Incision localization/marking” are made such asIncision Site Location Means based on Procedure Type and Point ofInterest. A means for marking the incision site for a surgical procedurebased on where the surgeon's point of interest is in the patient. Analgorithm is used to determine where the best incision site is on thepatient based on the procedure and where the surgeon's point of interestis. This process will make the incision site more accurate and theprocess for finding this site more repeatable, regardless of thepatient's anatomy. The amount of soft tissue damage that occurs insurgery will also decrease because the algorithm accounts for minimizingtissue damage.

Improvements in “Incision localization/marking” are made such as ImagingPort Location System using Artificial Intelligence. A system usesartificial intelligence to map where an imaging port should be locatedon the patient to most effectively map the patient's body. This systemconsiders where the surgeon is planning to make the initial incision onthe patient's body to help determine where the imaging port should belocated. The system re-evaluates where the imaging port should be placedduring different steps throughout the procedure.

Improvements in “Incision localization/marking” are made such asVirtualized Third Person Perspective of Endoscope Progress throughAR/VR. A method that virtualizes a third person perspective of endoscopeprogress through augmented reality or virtual reality means. The thirdperson perspective of the effort head would be mapped to other medicalimages used during surgery. This allows the camera point of view to bevirtualized, eliminating the need to have a second entry port. Thismethod comprising of a camera that is placed on the end effector itself,which provides a real-time image; and a tracking system shows theposition of the endoscope in the patient's body from the outside inreal-time. All this real-time data is overlaid on the pre-constructedmodel, which provides the surgeon with information that allows him orher to dynamically changed the perspective.

Improvements in “Incision localization/marking” are made such as UsingMRI Image for Robot Position Confirmation and Quantifying ConfirmationLevel. A system that enables the computer to analyze a pre-operative MRIimage using artificial intelligence to identify the patient'sabnormality. This information can be used to confirm the position of arobot. This would eliminate wrong level surgery. This is augmented witha method that quantifies the confirmation level of the robot's position,acting as a “confirmation meter”. This may include using many sources,such as multiple images at different levels, using pre-operative images,inter-operative images, computer-assisted navigation, and other means,to calculate the accuracy of the robot's position. The higher theposition accuracy, the higher the confirmation meter score.

Improvements in “Incision localization/marking” are made such as Methodof using an Endoscope as a Guide Wire. A method for designing theendoscope so that it also acts as a guide wire. This will allow theendoscope to constantly interact with the anterior-posterior (AP) view,allowing the surgeon to be constantly looking at the endoscope. Thissystem is expanded to cover the entirety of the procedure by using thesame functionality that allows the endoscope to function as a guide wireto locate the endoscope inside of the patient as an additional referencepoint for the surgical navigation program. The configuration of theendoscope can be selected based on the instrument to be delivered overit.

Improvements in “Initial Skin Incision” are made such as Use of WirelessEEG during Surgery to Track Patient Brain Activity. The use of awireless EEG system during surgery to track the patient's brainactivity. This would allow the surgeon to view the brain activity of thepatient during surgery to know how the patient is responding theoperation. For example, this would allow the surgeon to see the brainactivity of patient that is not completely unconscious from anesthesia.This could indicate when the patient is experiencing a large amount ofpain or excessive stimulation, signaling the surgeon to change tacticsor retract. This system could be improved with real-time imagerecognition and artificial intelligence identifying correlations betweenbrain activity patterns and adverse events during a procedure. Thismethod is detailed in this step as it is the first invasive step but isapplicable throughout the procedure.

Improvements in “Initial Skin Incision” are made such as Use of WirelessEEG on Patient for Medication during Surgery. The use of a wireless EEGsystem during surgery to help determine the amount of medication to givea patient. The EEG will track the amount of discomfort the patient isexperiencing. If the patient is experiencing a large amount ofdiscomfort or pain, the patient may be given more medication (i.e.anesthesia) during the surgery. Like 2a this system will require anartificial intelligence to examine, through a real-time imagerecognition system, the brain activity (e.g., brain activity patterns,brain activity spikes, etc.) and identify correlations with anesthesiabased adverse events. Pain, discomfort, patient vitals, etc. can bemonitored and evaluated to determine whether to modify the treatmentplan, administer anesthesia, etc. The AI/machine learning can be used toanalyze brain activity, patient feedback, or other patient parametersto, for example, improve safety, comfort, or the like. This method isdetailed in this step as it is the first invasive step but is applicablethroughout the procedure.

Improvements in “Initial Skin Incision” are made such as PatternRecognition from EEGs during Surgery for Pre-Operative Care Improvement.A method that analyzes information collected by wireless EEGs usedduring surgery to use to create optimal pre-operation procedures. Thedata collected by the EEG about the amount of pain or discomfortexperienced by the patient may be stored in a database and analyzed forpatterns based on various factors including medications given, thepatients' demographics, and other factors. Based on patterns found fromthe data, the type and amount of pre-operative medications given tofuture patients can be improved. This method focuses on the individual'sEEG patterns in response to stimulation prior to surgery rather than apopulation-wide correlation generator.

Improvements in “Muscle Fascia Incision” are made such as a Method ofMachine Learning to Identify Various Tissues. A method of machinelearning training for an AI surgical robot in which a surgeon identifiesthe different types of tissues (nerve, ligament, bone, etc.) and how touse different end effectors for each type of tissue. Rules can be addedto ensure that specific end effectors can only be used on specific typesof tissue (i.e. a drill is only used on bone, or a nerve is only touchedwith a probe or not allowed to be touched at all). This is applicable toall steps in the process but documented here as multiple tissue typesare involved in this specific step.

Improvements in “Muscle Fascia Incision” are made such as a NormalizeLighting for AI Image Recognition. A method of using a normalizedlighting for probe or imaging system for AI image recognition, inaddition, once the AI surgical robot can identify specific types oftissue a normalized lighting process will allow for the robot to see thesame or similar colors to easily identify previously learned tissues.

Improvements in “Muscle Fascia Incision” are made such as Mapping ofStimulated Muscles for Surgery. A method that allows for the location ofmuscles inpatient to be mapped through electric stimulation, producingan image that is used for surgery. This would enable a robotic surgeryto be guided with at least a first image of a patient's muscles.

Improvements in “Muscle Fascia Incision” are made such as ArtificialIntelligence System for Equipment Use in a Robotic Surgery. Anartificial intelligence system that uses information such as color,texture, and force to what equipment is being utilized in a roboticsurgery. For example, this system will understand when enough bone hasbeen worked through to recognize that the robot should stop using thedrill. This is like the concept described in the disclosure, but ratherthan relying solely on image recognition, the system incorporatescontact sensors, tissue type sensors (e.g., impedance sensors, opticalsensors, etc.), pressure sensors, force sensors, to improve the accuracyof the tissue identification system. The system can analyze signals fromthe sensors to determine, for example, the force required to continuethrough the tissue, tissue type, texture the tissue, or the like. Thesystem can perform procedures based, at least in part, on identifyingthe tissue type and its location.

As a drill or knife is robotically controlled, the drill or knife wouldhave highly sensitive force transducers. These force transducers producea real time X,Y,Z force set of data. The data is collected in manysuccessful operations. The real-time images not only have all theprevious metatags discussed, but also have the real time X,Y,Z forcedata. Now the system can be trained to show the delta force change goingfrom one tissue type to another. As above, the change in force in X,Y,Zcan be used to compare to real-time operations. If the tissues areidentified correctly and within range, and the forces and changes offorce are within range, the images are annotated with virtualinformation showing that tissues and forces and changes in force are inorder. If, however, the forces or changes of force appear out of normalrange, alarms would sound, and automated robotic stops would be done toinvestigate the out of norm situation. With this system, the surgeon cancreate a “sensitivity” of force change at various parts of theoperations, so the system may alarm when it approaches a nerve as theforce and change of force alarm is set at a more sensitive level thananother part of the operation.

Improvements in “Muscle Fascia Incision” are made such as Biomarkers forRobot Communication. A system that uses biomarkers to communicate with arobot where it is during surgery. This system can recognize what type oftissue the robot is touching and then be able to mark the tissueaccordingly. Using this system, a robot will be able to recognize whattype of tissues it is near and use that information to determine whereit is in the patient.

Improvements in “Docking of the retractor (tube) over the site ofinterest” are made such as AR/VR display of Surgical Device PlacementDuring Operation. A method for using AR or VR to display where asurgical device (i.e. a screw) is being inserted into the patient. Theprecise display of where the device should be located can be seen by thesurgeon during an operation, so the device is accurately placed. Thesurgical device placement recommendations are based upon the artificialintelligence's examination of surgical procedure data, patient data, andpostoperative outcomes, to identify correlations between deviceplacement and adverse events, or device placement and positivepost-operative outcomes.

Improvements in “Docking of the retractor (tube) over the site ofinterest” are made such as Vibrating the Robotic Retractor Tube. Aretractor tube that is a part of a robot that vibrates microscopicallyat a high speed. This would create a wavefront that would allow the tubeto insert into the patient's body with greater ease. This concept wouldbe augmented using the AI in conjunction with the image recognitionsystem to identify tissue types and adjust the vibrationfrequency/amplitude based upon correlations identified by the AI betweenvibration frequencies/amplitudes and positive outcomes/adverse events.

Improvements in “Docking of the retractor (tube) over the site ofinterest” are made such as Changing the Temperature of a Retractor Tube.The same as 4b, but a means for changing the temperature of theretractor tube (i.e. heating it up or cooling it down) instead ofvibration. This will allow the tube to insert into the patient's bodywith greater ease. This system will also benefit from the use of AI todetermine the correct temperature based on tissue type/patientattributes/procedure details/etc.

Improvements in “Confirming position/level prior to planned surgicalprocedure” are made such as Hand-Held Ball Tip Probe Mapping Device. Amethod of using a hand-held ball-tip probe with sensors located in therobotic arm/surgical tool to determine the position of the ball tipprobes location for creating a 3D map of a patient's spine to assist thesurgeon during the operation. The sensors could use haptic feedback todetermine if the ball tip probe was in contact with the spine (bone andcartilage) or soft tissue (muscles, etc.). This method is detailed inthis surgical step as an example but is applicable to each surgicalstep.

Improvements in “Confirming position/level prior to planned surgicalprocedure” are made such as Hand-Held Ball Tip Probe Mapping DeviceProviding Haptic Feedback to Surgeon. Similar to “Handheld ball-tipprobe mapping device”, this is a method of using a hand-held ball-tipprobe with sensors located in the robotic arm/surgical tool in order todetermine the position of the ball tip probes location for the purposeof creating a 3D map to assist the surgeon during the operation. therobotic provides haptic feedback that is transmitted to a surgeonhandheld device so that the surgeon can use the haptic feedback todetermine what the ball tip probe is in contact with. This method isdetailed in this surgical step as an example but is applicable to eachsurgical step.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Guidance adjustment based on surgeonadjustments in previous procedures based upon image recognition. Amethod of using image recognition to show a “point of view” from thedrill along the drill path. Using Artificial Intelligence pictures arebeing taken of the drill path and being compared to a historicaldatabase of similar surgeries/operations to optimize the drill path forthe least amount of potential damage and to improve the confidence ofthe surgeons using the tool. For example, during a typical decompressionsurgery 95% of surgeons move the drill two degrees to the left 10millimeters into the patient, the AI takes control of the robot andautomatically applies this movement. This method is detailed in thissurgical step as the example is for a decompression, but is applicableto each surgical step.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Machine Learning Method for GuidanceAdjustments on Surgery. A method of capturing the data from drillmounted “point of view” camera in which the data is uploaded into ahistorical database as it is collected and refined to improve theoperation of the system in future operations. If there is a knowncorrelation between the current surgery being performed and historicaldata the AI takes control of the robot to perform the movement. Thismethod is detailed in this surgical step as it is directly related to“Guidance adjustment based on surgeon adjustments in previous proceduresbased upon image recognition”, but is equally applicable in other stepsin a spinal surgery.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Method of Machine Learning for TactileSensation. A method of collecting data from pressure sensors on asurgical tool (i.e. drill) as well as collecting data from touch sensorsand using this data as context to inform the AI system of the surgicalrobot for the robot to learn a tactile sensation of the operation beingperformed. Surgical tools with more direct mechanical connectionsbetween the end effector and the surgeon's controls are monitored withsensors and imaging systems to determine the type and amplitude of thetactile feedback the surgeon receives during the manipulation ofdifferent tissue types during different actions. Those actions aremodeled and applied to robotic systems with shared control to providethe best feel for the surgeon while maintaining the precision advantagethat robotic systems provide over more traditional surgical tools. Thismethod is detailed in this surgical step as the removal of bone near thenerve tissue is one of the most important applications of thistechnology, but it is applicable in other steps in the spinal surgeryprocess.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Drill Data Collection for AI System. A methodof collecting data from a surgical drill such as RPMs, speed,resistance, and variations of these data points based on the movement ofthe drill. If the drill is in contact with bone, the data collectedwould be much different than if the drill was in contact with only softtissue. This data is collected and used to be incorporated into the AIsystem so that there is another data point for the AI system todetermine what type of tissue the drill is in contact with.

As above, where force and change in force in X,Y,Z are measured and usedto inform the surgeon, other robotic parameters such as (1) RPMs, (2)armature current, (3) angle and direction, (4) sound of motor, etc. aremeasured. As a drill or knife is robotically controlled, the drill orknife would have highly sensitive sensors for (1) RPMs, (2) armaturecurrent, (3) angle and direction, (4) sound of motor, etc. Theseparameters provide real-time robot set of data. These parameters providedata that is collected in many successful operations. The real-timeimages not only have all the previous metatags discussed, but also havethe real sensitive sensor data for (1) RPMs, (2) armature current, (3)angle and direction, (4) sound of motor, etc. Now the system can betrained to show the sensitive sensors changes for (1) RPMs, (2) armaturecurrent, (3) angle and direction, (4) sound of motor, etc. going fromone tissue type to another. As above the change sensitive sensors for(1) RPMs, (2) armature current, (3) angle and direction, (4) sound ofmotor, etc. can be used to compare to real-time operations. If thetissues are identified correctly and within range, and the sensitivesensors data for (1) RPMs, (2) armature current, (3) angle anddirection, (4) sound of motor, etc. and their associated changes arewithin range, the images are annotated with virtual information showingthat tissues and sensitive sensors data for (1) RPMs, (2) armaturecurrent, (3) angle and direction, (4) sound of motor, etc. are in order.If, however, the sensitive sensors data for (1) RPMs, (2) armaturecurrent, (3) angle and direction, (4) sound of motor, etc. or theassociated changes appear out of normal range, alarms would sound, andautomated robotic stops would be done to investigate the out of normsituation. With this system, the surgeon can create a “sensitivity” ofsensitive sensors data for (1) RPMs, (2) armature current, (3) angle anddirection, (4) sound of motor, etc. at various parts of the operations,so the system may alarm when it approaches a nerve as the sensitivesensors data for (1) RPMs, (2) armature current, (3) angle anddirection, (4) sound of motor, etc. and change of sensitive sensors dataalarm for (1) RPMs, (2) armature current, (3) angle and direction, (4)sound of motor, etc. is set at a more sensitive level than another partof the operation.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Vibration Cancelling High-Speed Drill. A methodof canceling out the haptic feedback caused by the movement of the drillso that a surgeon only receives haptic feedback from the body part thatthe drill is in contact with. Like noise cancellation technology used inheadphones, the drill can have similar vibrations incorporated when thedrill is being operated so that there no is no vibrational feedback tothe surgeon's hand. The remaining feedback from the operationalprocedure would be with the body parts that the drill is coming incontact with so that surgeon can better determine if the drill is stillin contact with bone or if it is in contact with the canal (or anothertype of tissue).

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Method of Measuring Using Shockwaves ThroughSurgical Drill. A method of using a seismograph on a surgical drill tosend a shockwave through the bone to determine the width or length ofthe bone to inform the surgeon (or AI drill) on how much bone is left todrill. This could also be used to determine how much of a specific bone(cancellous vs. cortical) is remaining to drill to inform the surgeon orthe AI drill.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Shared Laminectomy Procedure. A method of usingAI to inform a surgical robot drill to drill down to the lamina in alaminectomy and reach a predetermined point at which the surgical robotwill stop and the surgeon will resume the operation.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Laminectomy Using Ultrasound for AI SurgicalRobot where the AI surgical robot taking an image, it records anultrasound to determine if the canal has been reached.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Laminectomy Using Ultrasound for AI SurgicalRobot where the AI surgical robot taking an image, it records an CT todetermine if the canal has been reached.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Laminectomy Algorithm for AI Surgical RobotDrill Optimization. A variation would be to make variable both the drillspeed and the sampling rate that increases as proximity to the canalincreases.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Decompression Using AI Surgical Robot is usedfor laminotomies instead of laminectomies. Additional steps necessary toadjust the process will include further filtering of the patient datafrom the procedure database based on the volume of the lamina to beremoved. This will necessarily include correlation generation not juston the size of the patient's relevant anatomical structures, but alsocorrelations between post-surgical relief levels based upon the degreeof and location of lamina removal.

Improvements in “Decompression—removal of bone (lamina and/or facetjoint)” are made such as Mapping of Stimulated Nerves for Surgery is amethod that overlays a map of the patient's nerves on theanterior-posterior (AP) view. The nerves would be stimulated throughsome method (i.e. MEP, EMG); the nerve activity would be captured andrecorded by sensors placed on the patient. This would enable a roboticsurgery to be guided with at least a first image of an electricallystimulated spine. This concept is applicable to multiple steps in aspinal surgery but is described here as an additional method ofdetermining the distance the surgical robot needs to travel through thelamina.

Improvements in “Disc incision and disc removal” are made such as anUltrasonic Endoscopic Probe for Tissue Identification. A method thatuses an ultrasonic, endoscopic probe to determines the remaining tissuetype and location during surgery (i.e. discectomy). This will allowsurgeons to know what type of tissue is located herein the patientduring surgery. This is in addition to or in lieu of the imagerecognition system used in conjunction with AI to identify the tissuetype. The tissue's response to the ultrasonic endoscope is correlated inthe same way images are correlated with tissue type definitions.

Improvements in “Fusion—insertion of cage/bone graft” are made such as aFusion Using AI Surgical Robot. This invention improvement is used forspinal fusion. Additional steps necessary to adjust the process for thiswill include adjusting the imaging system placement to account for thegap between the end effector (that which is removing lamina in thelaminectomy procedure holding the bone graft in this step) and the endof the bone graft. Unlike the laminectomy where the imaging system isfocused on the point at which the end effector is in contact with thetissue, in this process the imaging system needs to be focused on wherethe leading end of the bone graft that is in contact with the tissue.The system will also need to adjust for the switch from tis sue removalin the laminectomy to tissue condition and bone graft progress throughthe desired path. A different, or additional area of focus in thissystem can be the graft surface and the graft site to check thecondition of the contact areas.

Improvements in “Fusion—insertion of cage/bone graft” are made such asSpinal Cage Placement with AI System. An artificial intelligence systemthat determines the optimal type, size, placement, and application angleof a spinal cage. This will increase accuracy and efficiency, as well asreduce the chance of complications from using the incorrect cage size orplacement. This concept combines the planning and cage selection stepsdetailed in the 3D printed screw guide disclosure with theinsertion/placement status steps in 8a.

Improvements in “Fusion—insertion of cage/bone graft” are made such asArtificial Intelligence for Push-Pull Test. An artificial intelligencesystem that uses force transducers to test the push-pull strength of thespinal cage during insertion. The system will be able to confirm theplacement of the spinal cage, reducing the risk of complicationspost-operatively. This embodiment is based on the push-pull sensorsbeing part of the surgical robot or a stand-alone tool. In 4d the forcetransducers are part of the cage.

Improvements in “Fusion—insertion of cage/bone graft” are made such asExpandable Spinal Cage with Force Transducers. A system that uses anexpandable cage that has multiple force transducers. These transducerscan measure force in all directions. The cage can automatically changeshape, size, and placement based on the feedback it is receiving fromthe force transducers. This system will ensure that the amount of forceexerted on each component of the change is correct.

Improvements in “Fusion—insertion of cage/bone graft” are made such asDisk Replacement Using AI Surgical Robot. This is a variation on thecage insertion concepts, in which the disk is replaced rather than thecage inserted.

Improvements in “Screw insertion” are made such as UltrasonicTransducers for Drilling Pilot Hole. This is a method that enables arobot to use ultrasonic transducers to measure its surroundings while itdrills the pilot hole for screw insertion. The robot will use anartificial intelligence system to analyze the feedback it receives fromthe transducers to ensure that there is a sufficient amount of bonesurrounding the pilot hole. If the transducers detect too little bonesurrounding the pilot hole, the drill will retract and adjust its angleappropriately to fix this problem. This will greatly reduce the risk ofa medial or lateral breach while drilling the pilot hole.

Improvements in “Screw insertion” are made such as Laser Projection ofPedicle Cross Section. This is a method of ensuring accurate screwplacement by projecting a laser target on the pedicle cross-section thatboth illuminates the point of insertion, but also the saddle directionnecessary for rod placement.

Improvements in “Rod insertion/screw placement” are made such asInstrumentation Using AI Surgical Robot. This improvement is orinstrumentation, such as rod and screw insertions/placement. Additionalsteps necessary to adjust the process for this will include adjustingthe imaging system placement to account for the gap between the endeffector (that which is removing lamina in the laminectomy procedure,but is holding the hardware in this step) and the end of the hardware.Unlike the laminectomy where the imaging system is focused on the pointat which the end effector is in contact with the tissue, in this processthe imaging system needs to be focused on where the leading end of thehardware is in contact with the tissue. The system will also need toadjust for the switch from tissue removal in the laminectomy to tissuecondition and hardware progress through the desired path. Wider field ofview imaging may be advantageous in this application to track thehardware's relative position to different tissues, in addition to or inlieu of, the point of contact imaging that is done in the laminectomy.For example, the system may strive to keep the rod being inserted aminimum distance from the skin surface while maneuvering it into itsfinal position.

Improvements in “Flexible Tube for Rod Placement” are made such as aFlexible Tube for Rod Placement which is a method for inserting aflexible tube into the patient to use as a guide for placing a rod. Theflexible tube has a working end; the direction that the working end ismoving can be changed by the surgeon. By using a flexible tube, it iseasier to connect all the U-shaped equipment in the patient.

Improvements in “Guidance and Navigation” are made such as a Nerveconduction for nerve location in robotic navigation which is a methodthat uses nerve conduction study to inform the robot of its proximity tothe nerve by sensing electrical conduction proximate to the endeffector.

Improvements in “Imaging” are made such as using Motion and SensorContext Data to Inform AI using a method of collecting sensor data froman operating room (i.e. visual, audio, or environmental data) andcollecting motion data from the tool being used to perform the surgery.The sensor data and motion data is used as context data to inform theArtificial Intelligence system of the robot to enhance the learning ofthe robot

Improvements in “Imaging” are made such as using Electric NerveStimulation for Diagnosing Nerve Injuries using a non-invasive methodfor finding injured nerves or disease in a patient using the stimulationof nerves with electricity. This method could replace theelectromyogram, which is considered a very painful surgery to findinjured nerves.

Improvements in “Imaging” are made such as using a Wireless EEG forDiagnosis by using a wireless EEG to diagnose diseases and disordersbased on patient's brain activity. This could be used to diagnosevarious diseases and disorders, including ADHD, depression, etc.

Improvements in “Imaging” are made such as using Wireless EEG Images inSpecific Steps for Surgical Procedures specifically implementing amethod for using imaging from a wireless EEG during specific steps in asurgical procedure in real-time. For example, imaging from a wirelessEEG may be used with x-ray images, bone position, and the position of adrill for specific steps throughout a procedure that involves insertinga screw into a patient's spine.

Improvements in “Imaging” are made such as using a Laser Overlay on APview for Incision Site Location specifically using a method for using alaser that overlays the location of the incision made by the surgeononto the anterior-posterior (AP) view. This invention places the opticallocation of the incision over the AP view, which can illustrate whetherthe incision should be changed (i.e. making it a bit longer) on acomputer. The laser could highlight on the patient's back where thechange (i.e. the lengthening) of the incision should be made.

Improvements in “invasiveness” are made such as using a Single Port withMultiple Arms, which uses a device that has a single port that expandsinto multiple arms when the robot is deployed inside of the abdomen.This allows for only one arm to be inserted into the patient's bodyinstead of having multiple arms.

Improvements in “planning and mapping” are made such as using a MachineLearning Workflow Optimization for Surgical Robot, using a method ofworkflow optimization by collecting data throughout a surgical processand then using the collected data for the following workflow. This datacan be incorporated into an AI system to provide surgical robot machinelearning capabilities on the surgical procedures.

Improvements in “planning and mapping” are made such as using a SurgicalPath Mapping for Neurosurgery, which provides a means for mapping thesurgical path for neurosurgery procedures that minimize damage throughartificial intelligence mapping. The software for artificialintelligence is trained to track the least destructive pathway. Theneurosurgeon makes the initial incision based on a laser marking on theskin that illuminates the optimal site. Next, a robot makes a tiny holeand inserts a guide wire that highlights the best pathway. This pathwayminimizes the amount of tissue damage that occurs during surgery.Mapping can also be used to identify one or more insertion pointsassociated with a surgical path. Mapping can be performed beforetreatment, during treatment, and/or after treatment. For example,pretreatment and posttreatment mapping can be compared by the surgeonand/or AI system. The comparison can be used to determine next steps ina procedure and/or further train the AI system.

Improvements in “post-operative spinal surgery” are made such as using aRestenosis Sensor in a method to insert a sensor into the patient thatmeasures pressure, bone growth, and/or force. The sensor would beinserted after a laminectomy and used to help determine if the patienthas restenosis based on the data it collects. This will increase theaccuracy of restenosis diagnosis.

Improvements in “post-operative spinal surgery” are made such as Robotfor Monitoring Surgery Recovery in the Epidural using a method thatenables a small robot to travel in the epidural space. This robot cancollect information pertaining to the patient's recovery process todetermine if the surgery was successful, and if not, what complicationsoccurred.

Improvements in “post-operative spinal surgery” are made such as aTemperature Sensor for Monitoring Post-Operative Recovery using a methodthat enables a biodegradable temperature sensor to be attached to animplant that was inserted into the patient during surgery. This sensorwill be able to detect the onset of post-operative infection during thefirst weeks of recovery. The sensor will naturally dissolve once theperiod of concern for infection is over.

Improvements in “post-operative spinal surgery” are made such as aBiodegradable, Biomarker Sensor for Detecting Infection using abiodegradable, biomarker sensor that can determine if the tissue becomesinfected post-operatively. If it detects an infection, a notification issent to the patient and/or the surgeon. This sensor is placed ontotissue that may become infected using a probe during surgery. More thanone sensor may be placed depending on how many areas of potentialinfection there are in the patient.

Improvements in “support surgical tools” are made such as a Ball Probewith an Accelerometer using a ball probe that has an accelerometer. Thisprobe can then collect information pertaining to force in areas ofinterest. For example, it can use the accelerometer data to measurebumps on a tissue surface and the AI determines when a threshold hasbeen crossed that the amplitude of the bumps indicates the surgery iscomplete

Improvements in “support surgical tools” are made such as a BiomolecularTagging of a Tumor using a method for bio molecularly tagging a tumorfor surgery. This includes a probe with biomarker on it to tag specifictypes of tissue (i.e. muscle). This mark will light up with LED when itis on the specified tissue type (i.e. muscle). The marker in this systemwill be able to recognize what type of tissue it is touching and then beable to mark it accordingly. This will improve robotic vision systemsduring a robotic surgery.

Improvements in “support surgical tools” are made such as a GrapheneProbe for Tagging Different Tissue Types, using a method that enablesgraphene to be incorporated into a probe that can mark different tissuetypes. This marker will recognize what type of tissue it is touch and beable to mark the tissue accordingly.

Improvements in “support surgical tools” are made such as a RodPlacement prior to Screw Placement using a method that involvesinserting the rod prior to inserting the screw. The rod is designed tohave a hole and phalanges. The entirety of the operation is done throughthe rod. When the screw is inserted into the rod, the rod is moved onescrew length to keep the rod solid. This allows the rod to be connectedto all the screws being inserted into the patient. This method allowsfor the rod's positioning to be optimized.

Improvements in “user interfaces in the spinal surgery invention” aremade such as a Robotic Scalpel with Voice-Command using a roboticscalpel that responds to voice-commands (i.e. “scalpel—2 millimeters”would move the scalpel 2 millimeters).

FIG. 6 illustrates a robotic system, according to an embodiment. Therobotic system can be used to perform the procedures disclosed herein.The robotic system can include one or more joints, links, grippers,motors, and effector interfaces, or the like. The configuration andfunctionality of the robotic system can be selected based on theprocedures to be performed. A robotic system with a high number ofdegrees of freedom can be used to perform complicated procedures whereasa robotic system with a low number of degrees of freedom can be used toperform simple procedures.

FIG. 7 illustrates end effectors, according to an embodiment. Theeffectors can be installed in the robotic system of FIG. 6 or otherrobotic systems disclosed herein. The end effectors can include, withoutlimitation, robotic grippers, cutting instruments (e.g., cutters,scalpels, or the like), drills, cannulas, reamers, rongeurs, scissors,clamps, or the like. The number and configuration of end effectors canbe selected based on the configuration of the robotic system and theprocedure to be performed. The AI system can select end effectors toperform one or more the steps in a surgical procedure.

In an illustrative embodiment, any of the operations, processes, etc.described herein can be implemented as computer-readable instructionsstored on a computer-readable medium. The computer-readable instructionscan be executed by a processor of a mobile unit, a network element,and/or any other computing device.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment,several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more processors(e.g., as one or more programs running on one or more microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and or firmwarewould be well within the skill of one of skill in the art in light ofthis disclosure. In addition, those skilled in the art will appreciatethat the mechanisms of the subject matter described herein are capableof being distributed as a program product in a variety of forms, andthat an illustrative embodiment of the subject matter described hereinapplies regardless of the particular type of signal bearing medium usedto actually carry out the distribution. Examples of a signal bearingmedium include, but are not limited to, the following: a recordable typemedium such as a floppy disk, a hard disk drive, a CD, a DVD, a digitaltape, a computer memory, etc.; and a transmission type medium such as adigital and/or an analog communication medium (e.g., a fiber opticcable, a waveguide, a wired communications link, a wirelesscommunication link, etc.).

Those skilled in the art will recognize that it is common within the artto describe devices and/or processes in the fashion set forth herein,and thereafter use engineering practices to integrate such describeddevices and/or processes into data processing systems. That is, at leasta portion of the devices and/or processes described herein can beintegrated into a data processing system via a reasonable amount ofexperimentation. Those having skill in the art will recognize that atypical data processing system generally includes one or more of asystem unit housing, a video display device, a memory such as volatileand non-volatile memory, processors such as microprocessors and digitalsignal processors, computational entities such as operating systems,drivers, graphical user interfaces, and applications programs, one ormore interaction devices, such as a touch pad or screen, and/or controlsystems including feedback loops and control motors (e.g., feedback forsensing position and/or velocity; control motors for moving and/oradjusting components and/or quantities). A typical data processingsystem may be implemented utilizing any suitable commercially availablecomponents, such as those typically found in datacomputing/communication and/or network computing/communication systems.

The herein described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely examples, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected”, or“operably coupled”, to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable”, to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

Various systems, methods, and techniques described above provide anumber of ways to carry out the invention. Of course, it is to beunderstood that not necessarily all objectives or advantages describedmay be achieved in accordance with any particular embodiment describedherein and may depend on the procedures to be performed, robotic system,and end effectors to be used. Thus, for example, those skilled in theart will recognize that the methods may be performed in a manner thatachieves or optimizes one advantage or group of advantages as taughtherein without necessarily achieving other objectives or advantages asmay be taught or suggested herein. Furthermore, the skilled artisan willrecognize the interchangeability of various features from differentembodiments disclosed herein and disclosed in U.S. App. No. 62/636,046;U.S. application Ser. No. 15/291,357; U.S. application Ser. No.16/001,055; U.S. application Ser. No. 16/012,464; U.S. application Ser.No. 16/015,486; U.S. application Ser. No. 16/023,014; and U.S.application Ser. No. 16/028,618, and all of these applications areincorporated herein by reference in their entireties. For example,guiding techniques, display techniques, endoscope control in theseapplications can be incorporated into the systems disclosed herein.Similarly, the various features and acts discussed above, as well asother known equivalents for each such feature or act, can be mixed andmatched by one of ordinary skill in this art to perform methods inaccordance with principles described herein.

From the foregoing, it will be appreciated that various embodiments ofthe present disclosure have been described herein for purposes ofillustration, and that various modifications may be made withoutdeparting from the scope and spirit of the present disclosure.Accordingly, the various embodiments disclosed herein are not intendedto be limiting.

1. A computer-implemented method for providing guidance during a roboticsurgical procedure upon a patient, the method comprising: detecting atissue from a visual image depicting an interior portion of the patientduring the surgical procedure using, at least in part, a real-timeobject detection system; and selecting, in response, at least in part,to the detecting the tissue, a tool for performing at least a portion ofa surgical step.
 2. The computer-implemented method of claim 1, whereinselecting the tool comprises: providing data associated with a type ofthe detected tissue to a machine learning system, the machine learningsystem trained to associate different tools with different types oftissue.
 3. The computer-implemented method of claim 2, wherein selectingthe tool comprises: applying at least one rule to confirm that theselected tool can be used with the detected tissue.
 4. Thecomputer-implemented method of claim 2, wherein the real-time objectdetection system comprises a neural network configured to: receive animage; divide the image into one or more regions; and predict boundariesand probabilities for each of the one or more regions.
 5. Thecomputer-implemented method of claim 1, wherein the method furthercomprises: providing a recommended placement position for the selectedtool.
 6. The computer-implemented method of claim 5, wherein the methodfurther comprises: causing a camera to focus upon an area to be workedon for the at least the portion of the surgical step.
 7. Thecomputer-implemented method of claim 5, wherein, the selected tool isnot within the patient at the time the tissue is detected from thevisual image, and wherein, after the selected tool is inserted into thepatient, the method further comprises: navigating the selected tool toan area to be worked on for the at least the portion of the surgicalstep.
 8. A computer system comprising: at least one processor; and atleast one memory, the at least one memory comprising instructionsconfigured to cause the computer system to perform a method forproviding guidance during a robotic surgical procedure upon a patient,the method comprising: detecting a tissue from a visual image depictingan interior portion of the patient during the surgical procedure using,at least in part, a real-time object detection system; and selecting, inresponse, at least in part, to the detecting the tissue, a tool forperforming at least a portion of a surgical step.
 9. The computer systemof claim 8, wherein selecting the tool comprises: providing dataassociated with a type of the detected tissue to a machine learningsystem, the machine learning system trained to associate different toolswith different types of tissue. tissue.
 10. The computer system of claim9, wherein selecting the tool comprises: applying at least one rule toconfirm that the selected tool can be used with the detected tissue. 11.The computer system of claim 9, wherein the real-time object detectionsystem comprises a neural network.
 12. The computer system of claim 8,wherein the method further comprises: providing a recommended placementposition for the selected tool.
 13. The computer system of claim 12,wherein the method further comprises: causing a camera to focus upon anarea to be worked on for the at least the portion of the surgical step.14. The computer system of claim 12, wherein, the selected tool is notwithin the patient at the time the tissue is detected from the visualimage, and wherein, after the selected tool is inserted into thepatient, the method further comprises: navigating the selected tool toan area to be worked on for the at least the portion of the surgicalstep.
 15. A computer system comprising: at least one processor; and atleast one memory, the at least one memory comprising instructionsconfigured to cause the computer system to perform a method forproviding guidance during a robotic surgical procedure upon a patient,the method comprising: detecting a tissue from a visual image depictingan interior portion of the patient during the surgical procedure using,at least in part, a real-time object detection system; selecting, inresponse, at least in part, to the detecting the tissue, a tool forperforming at least a portion of a surgical step; and causing a camerato focus upon an area to be worked on for the at least the portion ofthe surgical step.
 16. The computer system of claim 15, wherein thereal-time object detection system comprises a neural network configuredto: receive an image; divide the image into one or more regions; andpredict boundaries and probabilities for each of the one or moreregions.
 17. The computer system of claim 16, wherein selecting the toolcomprises: applying at least one rule to confirm that the selected toolcan be used with the detected tissue.
 18. The computer system of claim16, wherein selecting the tool comprises: providing data associated witha type of the detected tissue to a machine learning system, the machinelearning system trained to associate different tools with differenttypes of tissue.
 19. The computer system of claim 18, wherein, theselected tool is not within the patient at the time the tissue isdetected from the visual image, and wherein, after the selected tool isinserted into the patient, the method further comprises: navigating theselected tool to the area to be worked on for the at least the portionof the surgical step.
 20. The computer system of claim 18, wherein themethod further comprises: causing an indication where the tool should belocated to be displayed.